CONTENT-BASED MULTIMEDIA RETRIEVAL SYSTEM AND METHOD 
THEREOF 



BACKGROUND OF THE INVENTION 

5 

1. Field of the Invention 

Tine present invention relates to a system for retrieving multimedia data 
and a method thereof, in particular to a content-based multimedia retrieval system 
and a method thereof which is capable of retrieving multimedia data among 
10 different systems using color histograms constructed with different color spaces 
and color quantization methods. 

2. Description of the Prior Art 

In retrieval of content-based multimedia data in the conventional 
15 technology, color distribution of an image is displayed as a color histogram, the 
color histogram is defined in accordance with a color space and a color 
quantization method. 

In addition, in retrieval of content-based multimedia data in the 
conventional technology, the color histogram is mainly used. 
20 In other words, it is possible to perform an image or a video retrieval based 

on colors among systems using color histograms constructed with same color 
spaces and quantization methods on the internet circumstances. On the contrary, 
it is impossible to perform the image or video retrieval among systems using color 
histograms constructed with different color spaces and different quantization 
25 methods each other. 



Herein, the color histogram displays the color distribution of the image, is 
easy to calculate similarity for comparing the color histograms, has a character 
unconcerned in position of a color pixel (Sometimes, a character concerned in 
position of a color pixel is needed in an image retrieval. For that, other feature 
display is used.), and has a better performance than the other color display 
method (texture histogram) for content-based image retrieval. 

The content-based multimedia retrieval system using the color histogram 
will now be described with reference to accompanying FIG.1. 

FIG.1 is a block diagram illustrating the construction of the content-based 
multimedia retrieval system using color histograms in accordance with the 
conventional technology. 

As depicted in FIG.1, the content-based multimedia retrieval system using 
the color histogram comprises a color quantizer 100 for being inputted query 
image inputted by a user and extracting a color histogram as feature information of 
the query image, an image database 103 for storing a plurality of retrieval object 
image data 103-1 ~ 103-N, a feature database 102 for storing the color histograms 
102-1 ~ 102-N as the feature information of the plurality of retrieval object images, 
and a retrieval unit 101 for calculating similarity between the color histogram of the 
query image and the color histograms of the plurality of retrieval object images and 
outputting an image in accordance with the similarity as a retrieval result. 

The operation of the content-based multimedia retrieval system using the 
color histogram in accordance with the conventional technology will now be 
described. 

First, the color quantizer 100 extracts the color histograms 102-1 ~ 102-N 
in order to find the feature information of the image data 103-1 ~ 103-N about the 



plurality of retrieval object images. In other words, the color quantizer 100 extracts 
one color histogram 102-1 per each image 103-1. 

The extracted color histograms 102-1 ~ 102-N are connected to the 
pertinent images 103-1 ~ 103-N, and are stored on the feature database 102. In 
5 other words, the extracted color histograms 102-1 ~ 102-N stored on the feature 
database 102 display the color distribution of the each image. 

In addition, when the query image is inputted by the user, the color 
quantizer 100 extracts the color histogram about the query image. 

In other words, the color quantizer 100 quantizes the designated color 
10 space into a plurality of subordinate spaces in accordance with the designated 
color space and quantization method. 

Herein, the each quantized subordinate space is matched to one bin, color 
values of all pixels in a certain image constructs a color histogram in proportion to 
a frequency of occurrence corresponding to the each subordinate space. 
15 The retrieval unit 101 calculates the similarity of the color histograms 102- 

1 ~ 102-N displaying the color histogram of the extracted query image and the 
color distribution of the retrieval object images 103 ~ 103-N stored in the feature 
database 102, and outputs the retrieval result in accordance with the similarity. 

For example, the retrieval unit 101 calculates the similarity of the 
20 histograms 102-1 ~ 102-N extracted in advance for the image retrieval based on 
the color and stored in the feature database 102 and the histogram of the query 
image by using a known method, namely, a histogram matching method such as a 
histogram intersection, and outputs the each image corresponding to the color 
histogram as a retrieval result on the basis of the magnitude of the calculated 
25 similarity value. 
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Herein, when the color histogram of the query image Is extracted In 
advance and is stored in the feature database 102, the retrieval unit 101 calculates 
the similarity between the color histogram of the query image extracted in advance 
and the color histograms of the retrieval object images, and outputs an image in 
5 accordance with the similarity as a retrieval result (There is no need to extract 
newly the color histogram of the query image by the color quantizer.). 

On the contrary, when the query Image Is not stored on the feature 
database 102 after being extracted in advance, the color quantizer 100 extracts 
newly the color histogram of the query image, and provides it to the retrieval unit 
10 101. 

However, in the conventional content-based multimedia retrieval system 
using the color histogram, it is possible to compare the color histograms among 
different systems only when the same color space and same color quantization are 
adapted to the color quantizers. 

15 For example, when the color histograms constructed with same color 

space and same quantization method are used between same two systems on the 
internet circumstances, it is possible to perform the multimedia data retrieval 
based on the colors. 

On the contrary, between two different systems, when color histograms 

20 constructed with color spaces and quantization method are not same, it is 
impossible to perform the multimedia data retrieval based on the colors. It will now 
be described with reference to accompanying FIG. 2. 

FIG. 2 illustrates multimedia retrieval relations between different systems in 
accordance with the conventional technology. 

25 As depicted in FIG.2, because the meaning of each bin of the color 



histogram is changed by different color space and different color quantization, 
comparing a color histogram used in a local system A 200 with a color histogram 
used in a local system B is Impossible. 

For example, when color histograms 201-1 ~ 201 -N of images are stored 
in the feature database 201 of the local system A 200, a color histogram having 
125 (5x5x5) bin number is constructed by performing 5 levels of uniform 
quantization about each RGB element by using a RGB (Red Green Blue) color 
space 202. 

When color histograms 205-1 ~ 205-N of images are stored on the feature 
database 205, the local system B 204 constructs and stores a color histogram 
having 343 (7x7x7) bin number by performing 7 levels of uniform quantization 
about each HSV (Hue Saturation Value) element by using a HSV (Hue Saturation 
Value) color space 206. 

Accordingly, it is impossible to compare the color histogram of the local 
system A with the color system of the local system B. 

As described above, in the conventional technology, it is impossible to 
perform the multimedia data retrieval between the systems using the color 
histograms constructed with the different color spaces and color quantization 
methods each other. 

In addition, in the conventional technology, it is impossible to retrieve a 
retrieval object image about a query image of itself in a database of the other party 
between the systems using the color histograms constructed with the different 
color spaces and color quantization methods. 

In addition, in the conventional technology it is impossible to connect the 
databases which store various, different, wide range of images to a retrieval unit in 



a network circumstances such as the internet in order to retrieve images. 



SUMMARY OF THE INVENTION 

Accordingly, the object of the present invention is to provide a content- 
based multimedia retrieval system and a method thereof which is capable of 
performing a multimedia data retrieval among different systems using color 
histograms constructed with different color spaces and different color quantization 
methods. 

The other object of the present invention is to provide the content-based 
multimedia retrieval system and the method thereof which is capable of retrieving 
an image to be retrieved corresponding to a query image of itself in a database of 
the other party when retrieving multimedia data in systems using color histograms 
constructed with different color spaces and color quantization methods. 

The another object of the present invention is to provide the content-based 
multimedia retrieval system and the method thereof which is capable of retrieving 
images by connecting databases storing various, different, wide range of images 
to a retrieval unit on a network circumstances such as the internet. 

The another object of the present invention is to provide the content-based 
multimedia retrieval system and the method thereof which is capable of increasing 
multimedia data retrieval speed by extracting a color histogram of a query image in 
advance and storing the extracted color histogram of the query image. 

In order to achieve the objects of the present invention, the content-based 
multimedia retrieval system for retrieving content-based multimedia data 
comprises a first color quantization means for extracting a color histogram of query 



multimedia data; a second color quantization means for extracting color 
histograms of multimedia data to be retrieved ; and a histogram conversion means 
for converting the color histogram of the extracted query multimedia data and the 
color histogram of the multimedia data to be retrieved so as to be same each other. 

In order to achieve the objects of the present invention, the content-based 
multimedia retrieval method for retrieving multimedia data by comparing query 
multimedia data with multimedia data to be retrieved comprises inputting query 
multimedia data; converting the color histogram of the inputted query multimedia 
data and the color histogram of the multimedia data to be retrieved so as to be 
same each other; and calculating a similarity between the query multimedia data 
and multimedia data to be retrieved on the basis of the converted color histogram 
and outputting a retrieval result in accordance with the calculated similarity 

In order to achieve the objects of the present invention, the content-based 
multimedia retrieval method for retrieving multimedia data by comparing query 
multimedia data with retrieval object multimedia data comprises extracting a color 
histogram of the query multimedia data; extracting a color histogram of the 
multimedia data to be retrieved; comparing the extracted color space and color 
quantization method of the query image with the color space and color 
quantization method of the multimedia data to be retrieved ; and converting the 
color histogram of the extracted query multimedia data or the multimedia data to 
be retrieved into the same color histogram so as to have same color space and 
color quantization method each other when the color spaces and color 
quantization methods of the extracted multimedia data and multimedia data to be 
retrieved are different each other, and performing a retrieval in accordance with 
the similarity between the query multimedia data and multimedia data to be 



retrieved. 

In order to achieve the above-mentioned objects of the present invention, 
in the content-based multimedia retrieval system and the method thereof in 
accordance with the present invention, the multimedia data generation method 
comprises generating color space description information describing what color 
space the color histogram of the multimedia data is constructed; and generating 
quantization description information describing what color quantization method the 
color histogram of the multimedia data is constructed. 

In order to achieve the above-mentioned objects of the present invention, 
the histogram conversion method of multimedia data in a content-based 
multimedia retrieval on the basis of the color histogram as a feature element for 
content-based multimedia retrieval comprises projecting each subordinate color 
space of the color space of the quantized query multimedia data on the color 
space of the multimedia data to be retrieved; and distributing a bin value to each 
quantum of the color space of the multimedia data to be retrieved, which is 
overlapped with the projected color spaces. 

In order to achieve the objects of the present Invention, the content-based 
multimedia retrieval system comprises a first color quantization means for 
extracting color histogram of the multimedia data to be retrieved; a description 
means for describing color space and color quantization method of the extracted 
color histogram; and a second color quantization means for extracting color 
histogram of a query multimedia data with a method same as the described color 
space and color quantization method in order to perform the multimedia data 
retrieval. 

In order to achieve the above-mentioned objects of the present invention, 



in the retrieval method for retrieving tine multimedia data to be retrieved with an 
input of the query multimedia data, the content-based multimedia retrieval method 
comprises judging whether the color histogram of the query multimedia data 
corresponding to the color space and quantization method of the multimedia data 
5 to be retrieved is stored before; and calculating a similarity between the color 
histogram of the stored query multimedia data and the color histogram of the 
multimedia data to be retrieved and performing a multimedia retrieval in 
accordance with the calculated similarity. 



10 BRIEF DESCRIPTION OF THE DRAWINGS 



FIG.1 illustrates a construction of a content-based multimedia retrieval 
system using a color histogram in accordance with the conventional technology. 

FIG. 2 illustrates multimedia retrieval relations between different systems in 
15 accordance with the conventional technology. 

FIG. 3 illustrates a construction of a first embodiment of a content-based 
multimedia retrieval system in accordance with the present invention. 

FIG.4 illustrates multimedia retrieval relations between different systems in 
accordance with the present invention. 
20 FIG. 5 is a flow chart illustrating a first embodiment of a content-based 

multimedia retrieval method in accordance with the present invention. 

FIG.6 illustrates a color histogram conversion method in accordance with 
the first embodiment of the present invention. 

FIG. 7 is a flow chart illustrating a first embodiment of an operation process 
25 of a color histogram converter in accordance with the present invention. 
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FIG. 8 illustrates a color histogram conversion method using a sampling 
method in accordance with the present invention. 

FIG. 9 is a flow chart illustrating a second embodiment of an operation 
process of the color histogram converter in accordance with the present invention. 

FIG. 10 illustrates a second embodiment of the content-based multimedia 
retrieval system in accordance with the present invention. 

FIG. 11 is a flow chart illustrating a second embodiment of the content- 
based multimedia retrieval method in accordance with the present invention. 

FIG. 12 is a flow chart illustrating a third embodiment of the content-based 
multimedia retrieval method in accordance with the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

The present invention is to provide a content-based multimedia retrieval 
system and a method thereof which is capable of performing a multimedia data 
retrieval among different systems having multimedia data constructed with 
different color spaces and different color quantization methods (systems using 
color histograms constructed with different color spaces and color quantization 
methods). 

First, a first embodiment of the content-based multimedia retrieval system 
in accordance with the present invention will now be described in detail with 
reference to accompanying FIG. 3. 

FIG. 3 illustrates a construction of the first embodiment of the content- 
based multimedia retrieval system in accordance with the present invention. 

As depicted in FIG.3, the content-based multimedia retrieval system 



comprises a color quantizer 300 for extracting a color histogram of a query image 
after being inputted the query image by a user, a color space descriptor 301 for 
describing color space information which is basis of the extracted color histogram 
of the query image, a color quantization descriptor 302 for describing the color 

5 qunatization information which is basis of the extracted color histogram, a retrieval 
object system 305 for providing retrieval object image data 310-1 ~ 310-N and 
color histograms 309-1 ~ 309-N of the retrieval object images, a color histogram 
converter 303 for converting the color histogram of the extracted query image so 
as to correspond to the color histograms 309-1 ~ 309-N of the retrieval object 

10 images (multimedia data to be retrieved) in order to perform a multimedia retrieval 
among the different systems, and a retrieval unit 304 for calculating similarity 
among the color histogram of the converted query image and the color histograms 
309-1 ~ 309-N of the retrieval object images and outputting an image in 
accordance with the similarity as a retrieval result. 

15 The retrieval object system 305 comprises image databases 310 for 

storing the retrieval object images (multimedia data to be retrieved) (data image 1 
~ image N) 310-1 ~ 310-N, color quantizers 308 for extracting the color histograms 
309-1 ~ 309-N as the feature information of the retrieval object image data (Image 
1 ~ image N) 310-1 ~ 310-N, feature databases 309 for storing the extracted color 

20 histograms 309-1 ~ 309-N of the retrieval object images, color space descriptors 
306 for describing the color space information which is the basis of the color 
histogram of the retrieval object image, and color quantization descriptors 307 for 
describing the color quantization information which is the basis of the color 
histogram of the retrieval object image. 

25 Hereinafter, the operation of the content-based multimedia retrieval system 

11 



will now be described in detail. 

First, when the query image is inputted, the color quantizer 300 extracts 
the color histogram which is the feature information of the query image. 

Herein, the color histogram of the query image extracted by the quantizer 
5 300 is extracted as a certain color space and a certain color quantization. 

The color space quantizer 301 describes the color space of the extracted 
color histogram, and the color quantization descriptor 302 describes the color 
quantization method of the extracted color histogram. 

Meanwhile, the retrieval object system 305 comprises the feature 
10 databases 309 for storing the color histograms having multimedia data structure, 
namely, constructed with the same color spaces and same color quantization 
methods or different color spaces and different color quantization methods. 

Herein, the feature databases 309 are placed apart from each other, and 
are mainly connected to a network such as the internet. 
15 In addition, the retrieval object systems 305 to be the retrieval object (to 

own jointly image database or provide the image database to the retrieval unit) are 
systems for providing data on the internet circumstances (network), they can the 
same kind or different kind systems. 

Herein, the same kind of systems mean systems using color histograms 
20 constructed with same color spaces and color quantization methods, and the 
different kind systems mean systems using color histograms constructed with 
different color spaces and color quantization methods. 

The color quantizers 308 inside of the retrieval object system 305 extract 
the color histograms 309-1 ~ 309-N which are the feature information of the 
25 plurality of image data (Image 1 ~ Image N) 310-1 ~ 310-N stored in the image 
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database 310. In other words, the color quantizer 308 extracts the one color 
histogram 309-1 per the image data 310-1. 

The extracted color histogram 309-1 is connected to the pertinent Image 
310-1, and is stored In the feature database 309. 
5 Herein, the each feature database 309 stores the retrieval object images 

having the color histograms constructed with the different color spaces or different 
color quantization methods. 

The color space descriptors 306 describe the color spaces of the color 
histograms of the retrieval object images stored in the each feature database 309, 
10 and the color quantization descriptors 307 describe the qantlzations of the color 
histograms of the stored retrieval object Images. 

After that, the color histogram converter 303 finds out the color space and 
color quantization method (information) as the basis of the color histogram of the 
query image from the color space descriptor 301 and color quantization descriptor 
15 302. 

In addition, the color histogram converter 303 finds out the color space 
and color qunatization method as the basis of the color histograms of the retrieval 
object images from the color space descriptors 306 and color quantization 
descriptors 307 inside of the retrieval object system 305. 

20 After that, the color histogram converter 303 compares the color space 

and color quantization method (information) as the basis of the color histograms of 
the found retrieval object images with the color space and color quantization 
method (information) as the basis of the color histogram of the found query Image, 
and converts the color histogram of the query image so as to correspond to the 

25 color histograms of the retrieval object images when they are not same In the 
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comparing result. 

In addition, the color histogram converter 303 can perform a function for 
converting the color histograms of the retrieval object images so as to correspond 
to the color histogram of the query image. 

5 For example, whenever the color spaces and color quantization methods 

as the basis of the color histograms of the retrieval object images stored in the 
feature databases 309 are not same with the color space and color quantization 
method as the basis of the color histogram of the query image, the color histogram 
converter 303 converts the color histogram of the query image so as to correspond 

10 to the color histograms of the retrieval object images. 

Herein, when the color space and color quantization method of the color 
histogram of the query image is same with the color spaces and color quantization 
methods of the color histograms of the retrieval object Images (in the same kind 
system), the color histogram converter 303 provides the color histogram of the 

15 query image to the retrieval unit 307 as it is. 

On the contrary, when the color histogram of the query image and color 
histograms of the retrieval object images are constructed with the different color 
spaces and color quantization methods each other (in the different kind system), 
the color histogram converter 303 converts the color histogram of the query image 

20 (color space and color quantization method which are the basis of the color 
histogram of the query image) into the color space and color quantization method 
same with the color histograms of the retrieval object images (color spaces or 
color quantizations method which are the basis of the color histograms of the 
retrieval object images), and provides the converted color histogram of the query 

25 image to the retrieval unit 304. 
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As described above, the color histogram converter 303 converts one color 
histogram between the color histogram of the query image and color histogram of 
the retrieval object image so as to correspond to the construction (color space and 
color quantization) of the other color histogram in order to compare the color 
5 histogram of the query image with the color histograms of the retrieval object 
images. 

In other words, in retrieval of an image similar to the query image in the 
image database 310 storing the plurality of retrieval object images (multimedia 
data to be retrieved), the color histogram of the query image is compared with the 
10 color histograms 309-1 - 309-N of the image data stored in the feature database 
309. 

Herein, the color histogram converter 303 converts the color histogram of 
the query image and the color histogram of the retrieval object image into the 
same histograms in order to compare the color histogram of the query image with 

15 the color histogram of the retrieval object image. 

For example, when the user retrieves an image similar to the query image 
among the images to be retrieved constructed with the color histograms composed 
of different color spaces and different color quantization methods by different 
systems, whenever an image similar to the query image is retrieved, the color 

20 histogram of the query image is converted into the color histogram of the same as 
the color histogram of the image to be retrieved. Accordingly, the color histogram 
of the query image and the color histogram of the image to be retrieved can be 
compared. 

After that, the retrieval unit 304 outputs an image in accordance with the 
25 similarity as a retrieval result by comparing the color histogram of the query image 



or converted color histogram of the query image with the color histograms of the 
retrieval object images (color histograms of multimedia data to be retrieved) 
(calculating the similarity) on the color histogram converter 303. 

As described above, the content-based multimedia retrieval system in 
5 accordance with the present invention uses the color histogram converter 303, and 
can perform a multimedia retrieval between the different systems. 

FIG.4 illustrates a multimedia retrieval relations between different systems 
in accordance with the present invention. 

As depicted in FIG.4, the local system A 400 comprises feature databases 
10 401 for storing color histograms 401-1 ~ 401 -N of images, a color space descriptor 
402 for describing color space information corresponding to the color histograms 
401-1 ^ 401-N, and a color quantization descriptor 403 for describing color 
quantization Information. 

The local system B 404 comprises feature databases 405 for storing color 
15 histograms 405-1 ~ 405-N of images, a color space descriptor 406 for describing 
color space information corresponding to the color histograms 405-1 ~ 405-N, and 
a color quantization descriptor 407 for describing color quantization information. 

Hereinafter, the multimedia retrieval between the local systems A 400 and 
B 404 (different systems) will now be described in detail. 
20 First, the color space descriptor 402 inside of the local system A 400 

describes the color space which is the basis of the color histogram as a RGB color 
space, and the color quantization descriptor 403 describes 5 levels of uniform 
quantization of the RGB color space. 

On the contrary, the color space descriptor 406 inside of the local system 
25 B 404 describes the color space which is the basis of the color histograms as a 

16 



HSV color space, and the color quantization descriptor 407 describes 7 levels of 
uniform quantization of the HSV color space. 

Accordingly, when the comparison (retrieval) object image about the query 
image of the local system A is retrieved in the feature database 405 of the local 
5 system B 404, the local system A 400 can recognize the local system B 404 is 
different kind of system through the color space descriptors 402, 406 and color 
quantization descriptors 403, 407. 

The local system A 400 can recognize the local system B is different kind 
of system, the color histogram (corresponding to the query image) based on the 

10 RGB color space (each 5 level uniform quantization) is converted into the color 
histogram (corresponding to the retrieval object image) based on the HSV color 
space (each 7 level uniform quantization), the converted color histogram is 
compared with the color histogram based on the HSV color space (each 7 level 
uniform quantization), accordingly the multimedia data retrieval is possible. 

15 In other words, in order to compare the color histogram used in the local 

system A 400 with the color histogram used in the local system B 404, the one 
color histogram (color space and color quantization) between them is converted so 
as to correspond to the construction of the other color histogram (color space and 
color quantization), accordingly the multimedia data retrieval is possible. 

20 In performing the above-described color histogram conversion, one 

quantum of the RGB color space can be projected to the HSV color space, and 
can be overlapped with at least one quantized subordinate space of the HSV color 
space. Herein, the degree of the projected color space can be found relatively and 
accurately by an integral calculus, however the calculation is complicate, 

25 accordingly it is not advisable in embodiment of a real-time system. 
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In order to solve above-mentioned problem, in the present invention, a 
histogram is converted quickly by using a sampling method, it will be described in 
detail later. 

Hereinafter, when the extract method of the color histogram of the query 
5 image and the color histogram of the retrieval object image is different in 
accordance with the color space and color quantization, the similarity is calculated 
by comparing the two color histograms, it is judged whether the query image is 
similar to the retrieval object image in accordance with the calculated similarity, the 
above-described content-based multimedia retrieval method will now be described 
10 in detail with reference to accompanying FIG. 5. 

FIG. 5 is a flow chart illustrating the first embodiment of a content-based 
multimedia retrieval method in accordance with the present invention. 

First, when the query image is inputted to the multimedia retrieval system 
S500, it is retrieved whether the color histogram of the query image which is same 
15 with the color histogram of the inputted query image is extracted before S501 . 

When there is the color histogram extracted before, the color histogram 
value extracted before is read from the feature database S502. 

On the contrary, when the query is made newly by the user or the query 
image is made from the other image data and there is no color histogram 
20 extracted before, the color histogram about the query image inputted through the 
color quantizer 300 is extracted S503. 

In the method for extracting the color histogram of the inputted query 
image, all pixels of the image are converted into determined color space values, 
the converted color space values are divided into a plurality of subordinate color 
25 spaces in accordance with the color quantization, and it is recognized which 
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subordinate color space transforms (displays) the each converted pixel value. 

After performing the process to the all pixels, it is possible to recognize 
how many pixels are projected to which subordinate color space. In other words, it 
is possible to know the values of the transofrmed bin number, and values found by 
arranging the projected bin numbers with the first dimension vector is the color 
histogram. 

Herein, the each first vector element about the values arranged with the 
first vector is "Histogram Bin", and it means the subordinate color space. 

Accordingly, the bin number of the color histogram coincides with the color 
quantization number. 

After that, the color histogram is normalized regardless of the image size 
by dividing the first dimension vector value by a total image pixel number 

And, the color histograms values of the retrieval object images stored in 
the feature database are read S504. 

Next, the color histogram of the query image extracted or read in the 
above process S502, S503 are compared with the color histograms of the all 
retrieval object images stored in the feature database. 

In other words, the color space and color quantization method as the basis 
of the color histogram of the query image is compared with the color spaces and 
color quantization methods of the color histograms of the all retrieval object 
images S505. 

When the color space and color quantization method as the basis of the 
color histogram of the query image is compared with the color spaces and color 
quantization methods of the color histograms of the all retrieval object images, and 
they are not same, the histogram converter 303 converts the color histogram of 
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the query image so as to correspond to the color histogram of the retrieval object 
image. 

After that, the converted color histogram of the query image is compared 
with the color histogram of the retrieval object image in order to calculate the 
similarity S507. 

On the contrary, when the color space and color quantization method as 
the basis of the color histogram of the query image is compared with the color 
spaces and color quantization methods of the color histograms of the all retrieval 
object images and they are same, the similarity is calculated S507. 

Herein, as the method for calculating the similarity the known methods 
can be used. 

In addition, the similarity is compared with a certain threshold value S508, 
and when the similarity is bigger than the certain threshold value, an image which 
corresponds to the color histogram of the image to be retrieved is regarded as the 
similar image and is outputted S509. 

After that, a retrieval about the one query image is performed by 
performing the calculating process S507 for comparing and calculating the 
similarity of the color histograms of the all retrieval object images, when the color 
histogram of the present retrieval object image which is compared and calculated 
the similarity presently is not the color histogram of the last retrieval object image 
S510, the processes from the reading process S504 for reading the color 
histogram value of the retrieval object image stored in the feature database are 
performed repeatedly. 

After that, when the calculating process for calculating the similarity is 
finished up to the color histogram of the last retrieval object image, it is judged 
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whether any other query image is inputted S511, when the other query image is 
inputted, the processes from the query image inputting process S500 are 
performed repeatedly. 

Meanwhile, the color histogram converting method in accordance with the 
present invention transforms the each subordinate space (each quantum) of the 
color space of the quantized query image on the color space of the retrieval object 
image in order to convert the color histogram, and distributes the bin value on the 
proportion to the overlapped volume. The above-mentioned process will now be 
described in detail with reference to accompanying FIG. 6. 

FIG. 6 illustrates a color histogram conversion method in accordance with 
the first embodiment of the present invention. 

As depicted in FIG.6, the color space of the query image is described as 
the RGB color space 600, the RGB color space is quantized by dividing into 3 
pieces about each component (RGB). In other words, 27 numbers of quantized 
subordinate spaces exist in the RGB color space, and the query image is 
described as the color histogram having 27 bin size. 

In addition, the color space of the retrieval object image is described as 
the HSV color space 601, it is separately divided into 3 pieces about the S and V 
components, the H is divided into 8 pieces or 6 pieces in accordance with region 
of a S value and is not divided in region adjacent to an achromatic color. In other 
words, there is 45 numbers of quantized subordinate spaces in the HSV color 
space, and the retrieval object images are described as a color histogram having 
45 bin size. 

One quantum in the RGB color space is projected to the HSV color space, 
and can be overlapped with the at least one quantized subordinate space of the 



HSV color space. 

In addition, a quantized subordinate space QSS_1 600A of the RGB color 
space 600 is projected along quantized subordinate spaces TQSS_1, TQSS_2, 
TQSS_9 of the HSV color space 601 601 A. 

Herein, a reference numeral 603 is a horizontal cross section of the HSV 
color space 601, and a reference numeral 602 is a vertical cross section of the 
HSV color space 601. 

Herein, when each volumetric ratio of the portion overlapped v\/ith the 
TQSS_1, TQSS_2, TQSS_9 by the projected quantized subordinate space QSS_1 
600A is 0.54, 0.35, 0.11, a histogram value H_Q (QSS_1) of the quantized 
subordinate space QSS_1 (600A) is separately projected to histogram values H_T 
(TQSS_1), H_T (TQSS_2), H_T (TQSS_9) of the TQSS_1, TQSS_2, TQSS_9 as 
much as 0.54 x H_Q (QSS_1), 0.35 x H_Q (QSS_1), 0.11 x H_Q (QSS_1), 

In other words, the H_T (TQSS_1), H_T (TQSS_2), H_T (TQSS_9) are 
renewed as below Equation 1~3. 

[Equation 1] 

H_T(TQSS_1):=H_T(TQSS_1) + H_Q(QSS_1) x 0.54 
[Equation 2] 

H_T(TQSS_2):=H_T(TQSS_2) + H_Q(QSS_1) x 0.35 
[Equation 3] 

H_T (TQSS_9) := H_T (TQSS_9) + H_Q (QSS_1) x 0.11 
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Herein, the H_T means a color histogram of the normalized retrieval object 
image, and the H_Q means a color histogram of the normalized query image. 

In addition, the H_T (TQSS_k) describes a histogram value corresponding 
to the index TQSS_k of the H_T, and the H_Q (QSSJ) describes a histogram 
5 value corresponding to the index QSSJ of the H_Q. 

When the above-mentioned process is performed about the all quantized 
subordinate space QSSJ of the RGB color space, the H_T (TQSS_k) values are 
renewed gradually, a histogram of the H_T (TQSS_k) found after the renewal is 
the color histogram which quantizes the HSV color space into 45 color grades 
10 (above-described quantization method) (Herein, I = 1, 2, , 26, 27 and k = 1 , 2, , 44, 
45). The above-described color histogram conversion process will now be 
described in detail with reference to accompanying FIG. 7. 

FIG. 7 is a flow chart illustrating a first embodiment of an operation process 
of the color histogram converter in accordance with the present invention. 
15 First, when the color spaces and quantization methods of the query image 

and retrieval object image are different each other, the conversion of the color 
histogram starts S700. 

After that, the all subordinate spaces QSSJ defined through the color 
space and quantization method as the basis of the color histogram of the query 
20 image are considered S701 . 

In other words, the all subordinate spaces QSSJ are projected to the color 
space of the retrieval object image, and a volumetric value VJ of the projected 
solid PV (QSSJ) is calculated S702. 

In addition, it is checked whether the all subordinate spaces QSSJ are 
25 overlapped with the subordinate spaces TQSS_k defined by the quantization 
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method of the retrieval object image, when they are overlapped, a volumetric value 
IV_I<J of a solid IV overlapped with the subordinate spaces TQSS_k and the 
projected solid PV (QSSJ) is calculated S703. 

After that, the all subordinate spaces TQSS_k including the overlapped 
volume among the quantized subordinate spaces of the color space of the retrieval 
object image are considered S704. 

In the color histogram of the retrieval object image and color histogram of 
the query image, the all subordinate spaces TQSS_k and all subordinate spaces 
QSSJ are separately renewed by below Equation 4 S705, accordingly the color 
histogram conversion is finished. 

[Equation 4] 

H_T (TQSS_k) : = H_T (TQSS_k) + H_Q (QSS_i) x IV_k_i / V_i 

Herein, the IV_ kj is the volumetric value of the solid IV overlapped with 
the projected solid PV (QSS_I), the V_i is a volumetric value of the solid PV 
(QSS_i) which is projected to the color space of the retrieval object image. 

After that, when the renewal process is not performed up to a subordinate 
space TQSS_k among the all subordinate spaces TQSS_k S706, the processes 
from the considering process for considering the all subordinate spaces TQSS_k 
S704 are performed repeatedly. 

In addition, when the renewal process is not performed to the last 
subordinate space QSSJ among the all subordinate spaces QSSJ S707, the 
processes after the considering process S701 for considering the all subordinate 
spaces QSS-i are performed repeatedly. 
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Herein, a volumetric value of the projected solid can be found relatively 
and accurately by an integral calculus, however the calculation is complicate, 
accordingly it is difficult to adapt to a real-time system. 

In order to solve the problem, in the present invention, the color histogram 
5 can be converted quickly by finding a volumetric value of the solid approximately 
by using a sampling method. 

The histogram conversion method using the sampling method will now be 
described in detail with reference to accompanying FIG. 8. 

As depicted in FIG. 8, 10 numbers of sampling points 800A are uniformly 
10 extracted from the subordinate space QSS_1 BOOB of the RGB color space 800. 

In other words, the transformation degree is calculated by finding out in 
which subordinate space of the HSV color space 802 the extracted 10 numbers of 
sampling modes exist. 

Meanwhile, a reference numeral 801A describes a total volume projected 
15 the subordinate space QSS_1 800B, and a reference numeral 801 B describes the 
sampling points 800A are projected to the HSV color space. 

in addition, a reference numeral 803 describes a horizontal cross section 
of the HSV color space 801, and a reference numeral 802 describes a vertical 
cross section of the HSV color space 801 . 
20 Whenever the each sampling point 800A is projected to the HSV color 

space 801, it is projected to the pertinent subordinate space as much as 0.1 x 
H_Q (QSS_1), when it is performed to the all 10 sampling points 800A, 5 sampling 
points are projected to the TQSS_1 as much as 0.5 x H_Q (QSS_1), it is 
projected to the TQSS_2 as much as 0.4 x H_Q (QSS_1), and it is projected to 
25 theTQSS_9asmuchas0.1 x H_Q (QSS_1). 
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In other words, the H_T (TQSS_1), H_T (TQSS_2), H_T (TQSS_9) are 
renewed as below Equation 5-7. 

[Equation 5] 

5 H_T(TQSS_1):=H_T(TQSS_1) + H_Q(QSS_1) x 0.5 
[Equation 6] 

H_T (TQSS_2) := H_T (TQSS_2) + H_Q (QSS_1) x 0.4 

10 [Equation 7] 

H_T (TQSS_9) := H_T (TQSS_9) + H_Q (QSS_1) x 0.1 

Herein, the H_T is the color histogram of the normalized retrieval object 

image, and the H_Q is the color histogram of the normalized query image. 
15 In addition, the H_T (TQSS_k) is the histogram value corresponding to the 

index TQSS_k of the H_T, and the H_Q (QSSJ) is the histogram value 

corresponding to the index QSSJ of the H_Q. 

When the above-mentioned processes are performed to the all quantized 

subordinate space QSSJ of the RGB color space, the H_T (TQSS_k) values are 
20 renewed gradually, the histogram of the H_T (TQSS_k) found after the renewal is 

the color histogram which quantizes the HSV color space into 45 color grades 

(above-described quantization method) (Herein, i = 1, 2, , 26, 27 and k = 1, 2, , 44, 

45). 

In the present invention, in sampling of the quantized subordinate space of 
25 the color space of the query image, the number of the sampling point can be 
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different in accordance with Importance of the each subordinate space. 

In other words, by differentiating the number of the sampling point in 
accordance with importance of the each subordinate space, the present invention 
can improve the accuracy in conversion of the histogram, and at the same time 
5 can decrease the histogram conversion time. 

Herein, the importance of the subordinate space is the bin value of the 
histogram corresponding to the subordinate space and the volumetric value of the 
subordinate space. 

First, in sampling of the subordinate space, the pertinent bin value is 
10 inputted from the color histogram, when the bin value is big, lots numbers of the 
sampling points are made, when the bin value is small, fewer numbers of the 
sampling points are made. 

Because the subordinate space having 0 histogram bin value do not 
contribute to the histogram conversion, there is no need to perform the sampling, 
15 in particular, in the histogram of the minutely quantized color space there is pretty 
many beans having 0 bin value, accordingly it is possible to decrease the 
histogram conversion time. 

In the bin value excluding 0, the reflection degree of the volumetric value 
of the quantized subordinate space of the color space of the query image is 
20 adjusted in proportion to the bin value. 

In other words, as larger as the volume of the quantized subordinate 
space of the color space of the query image, lots of the sampling points are 
needed in order to cover the space, in sampling of the quantized subordinate 
space of the color space of the query Image, the number of the sampling points is 
25 determined in consideration of the volumetric value of the subordinate space and 



bin value at the same time. 

However, when the importance of the subordinate space is determined in 
order to determine the number of the sampling points, the present invention has 
no limit for a method considering the volumetric value and bin value. 
5 The overall histogram conversion process including the determining 

process for determining the number of the sampling points of the quantized 
subordinate space of the color space of the query image in accordance with the 
importance of the volumetric value and bin value will now be described in detail 
with reference to accompanying FIG. 9. 
10 FIG. 9 is a flow chart illustrating a second embodiment of an operation 

process of the color histogram converter in accordance with the present invention. 

First, when the color spaces and quantization methods of the query image 
and retrieval object image are different each other, the conversion of the color 
histogram starts S900. 
15 In order to convert the color histogram, the all quantized subordinate 

spaces of the color histogram to be converted are considered S901. 

In other words, the number of the sampling point (No_of_SP) about the 
quantized subordinate space QSS_i of the color space of the query image is 
determined in proportion to the histogram bin value and volumetric value of the 
20 subordinate space QSSJ S902, and the sampling points are evenly extracted as 
same as the determined number from the subordinate space QSSJ S903. 

Herein, the each sampling point is described as a point value of the 
applied color space (RGB or HSV). 

In other words, when the RGB color space is applied, the sampling point is 
25 described as R=1 50, G=1 30, B=1 20. 
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In addition, after considering the extracted all sampling points (SPJ), it is 
compared whether the color space as the basis of the color histogram of the query 
image (namely, the color space for extracting the sampling point) and the color 
space as the basis of the color histogram of the retrieval object image (the color 
5 space about the color histogram to be converted) are same S905. 

When the color space as the basis of the color histogram of the query 
image and the color space as the basis of the color histogram of the retrieval 
object image do not correspond, the color value of each sampling point SPJ of the 
all extracted sampling points is converted into the color space value of the retrieval 
10 object image S906. 

On the contrary, when the color space as the basis of the color histogram 
of the query image and the color space as the basis of the color histogram of the 
retrieval object image correspond, because only the color quantization method is 
different, the conversion process S906 for converting the color value into the color 
15 space value of the retrieval object image is abridged. 

After that, the extracted all sampling points SPJ are projected to the color 
space of the retrieval object image, it is checked in which qunatized subordinate 
space TQSS of the color space of the retrieval object image the projected value is 
included S907, and the color histogram bin value corresponding to the H_T 
20 (TQSS_k) is renewed as below Equation 8 S908. 

[Equation 8] 

H_T (TQSS_k) := H_T (TQSS_k) + H_G (QSS_i) x 1 / No_of_SP 
25 Herein, the H_T means the color histogram of the normalized retrieval 
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object image, and the H_Q means the color histogram of the normalized query 
image. 

In addition, the H_T (TQSS_I<) means the histogram value corresponding 
to the index TQSS_k of the H_T, and the H_Q (QSSJ) means the histogram value 
corresponding to the index QSSJ of the H_Q. 

After that, when the sampling point SPJ which does not perform the 
renewal process S908 remains 8909, the processes from the considering process 
8904 for considering the extracted all sampling points SPJ are performed 
repeatedly. 

After that, when the quantized subordinate space QSSJ which does not 
perform the renewal process S908 remains S910, the processes from the 
considering process 8901 for considering the all quantized subordinate spaces are 
performed repeatedly. 

Accordingly, the color histogram can be converted by performing the 
above-described processes about the extracted all sampling points SPJ and all 
quantized subordinate spaces QSSJ. 

In an image retrieval, the color histogram conversion method using the 
above-described sampling method can convert the color histogram into the 
different color space and different quantization, and it is adaptable when there is 
need to convert the histogram having the different quantization method into the 
other quantization method about elements constructing not only the color but also 
a certain at least first dimension space. 

Hereinafter, the other embodiment of the content-based multimedia 
retrieval system and method thereof will now be described in detail with reference 
to accompanying FIG. 10 ~ 12. 
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FIG. 10 illustrates a second embodiment of the content-based multimedia 
retrieval system in accordance with the present invention. In other words, the color 
histogram converter 303 is not comprised in the second embodiment of the 
content-based multimedia retrieval system, and a color histogram database 1001 
of the query image is comprised. 

As described above, the content-based multimedia retrieval system 
comprises a color quantizer 1000 for extracting the color histogram of the query 
image by being inputted the query image from the user, a color histogram 
database 1001 of the query image for storing the extracted color histogram, a 
retrieval object system 1003 (same as the 305 of FIG. 3) for providing the retrieval 
object image data 1007-1 ~ 1007-N and the color histograms 1008-1 ~ 1008-N of 
the retrieval object images, and a retrieval unit 1002 for calculating the similarity 
between the color histogram of the stored query image and color histogram of the 
retrieval object image and outputting an image according to the similarity as a 
retrieval result when the color histogram of the query image same with the inputted 
query image is quantized and extracted with the color space and quantization 
method corresponding to the color histogram construction of the retrieval object 
image, and is stored on the color histogram database 100 of the query image. The 
operation of the content-based multimedia retrieval system will now be described 
in detail. 

First, when the query image is inputted, the color quantizer 1000 extracts 
the color histogram as the feature information of the query image, and stores the 
extracted color histogram on the color histogram database 1001 of the query 
image. 

Herein, the color quantizer 1000 references the color space and 



quantization method as the basis of the retrieval object image from the retrieval 
object system 1003, and quantizes the color space and quantization method as 
the basis of the color histogram of the extracted query image so as to correspond 
to the color space and quantization method of the retrieval object image. 

Accordingly, although the query image is same, the color histograms 1001- 
1 ~ 1001-N constructed with the different color spaces and quantization methods 
can be stored in the color histograms 1001-1 ~ 1001-N. 

Meanwhile, the present invention can extract the color histogram more 
quickly by adapting a method for finding the color histogram by reducing the size 
of the query image as a thumbnail image form. 

In addition, the quantization of the query image is performed in the 
retrieval object system 1003, and it is also possible to compare and retrieval of the 
quantization result (color histogram information). 

Herein, the retrieval object system 1003 to be the retrieval object (owning 
jointly the image database or providing the image database to the retrieval unit) is 
a system providing data on the internet circumstances (network), it can be same 
kind or different kind system. 

The color quantizers 1006 inside of the retrieval object system 1003 
extracts color histograms 1008-1 ~ 1008-N as the feature information of a plurality 
of image data (Image 1.... Image N) 1007-1 ~ 1007-N stored in the image 
database 1007. In other words, they extracts the one color histogram 1008-1 per 
the one image 1007. 

Herein, the extracted color histograms 1008-1 ~ 1008-N are connected to 
the pertinent images 1007-1 ~ 1007-N, and are stored in the feature database 
1008. 
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In addition, the retrieval object system 1003 comprises a color quantizer 
1006 for extracting the color histogram of the retrieval object image, color space 
descriptors 1004 for describing the color space information as the basis of the 
color histogram stored in the feature database 1008 after being extracted by the 
color quantizer 1006, and color quantization descriptors 1005 for describing the 
color quantization information as the basis of the stored color histogram. 

In addition, the retrieval object system 1003 is constructed with the 
plurality of feature databases 1008, images having color histograms constructed 
with different color spaces and different color quantizations are stored in the each 
feature database 1008, the color space and quantization of the color histogram 
can be described by color space quantizers 1004 and color quantization 
descriptors 1005. 

Herein, the each feature database 1008 is placed apart from each other, 
and is connected with a network such as the internet. 

After that, the retrieval unit 1002 judges whether the color histogram of the 
query image same with the query image inputted by the user is quantized by the 
color space and quantization method corresponding to the color histogram 
construction of the retrieval object image, is extracted, and is stored in the color 
histogram database 1001 of the query image. 

In the judging process, when the color histogram of the query image same 
with the query image inputted by the user is stored in the color histogram database 
1001 of the query image, the retrieval unit 1002 calculates similarity between the 
color histogram of the stored query image and color histogram of the retrieval 
object image, and outputs an image in accordance with the similarity as a retrieval 
result. 
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On the contrary, when the color histogram of the query image same with 
the query image inputted by the user is not stored in the color histogram database 
1001 of the query image, the retrieval unit 1002 extracts the color histogram of the 
inputted query image, calculates similarity between the color histogram of the 
5 stored query image and color histogram of the retrieval object image, and outputs 
an image in accordance with the similarity as a retrieval result. 

Herein, the color histogram of the extracted query image is stored in the 
color histogram database 1001 of the query image. 

Hereinafter, the second embodiment of the content-based multimedia 
10 retrieval method performed by the content-based multimedia retrieval system of 
FIG. 10 will now be described in detail with reference to accompanying FIG. 11. 

FIG. 11 is a flow chart illustrating the second embodiment of the content- 
based multimedia retrieval method in accordance with the present invention. 

First, when the query Image is inputted S1 100, the quantizer 1000 (or 
15 retrieval unit 1002) acquires the color space description information and 
quantization description information of the retrieval object Image S1 101. 

After that, the quantizer 1000 (or retrieval unit 1002) extracts the color 
histogram of the query image by quantizing the color histogram of the query image 
as the color space and quantization method corresponding to the color histogram 
20 construction of the retrieval object image by referencing the color space 
description information and quantization description information S1 102. 

After that, the color histogram values 1008-1 ~ 1008-N of the retrieval 
object images stored in the feature database 1008 are read S1 103, the similarity 
between the extracted color histogram of the query image and color histogram of 
25 the read retrieval object image are compared S1 104. 
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When the similarity of the color histogram of the query image and color 
histogram of the retrieval object image is larger than a certain threshold value 
S1105, the retrieval object image is outputted as a similar image S1106. 

On the contrary, when the similarity of the color histogram of the query 
5 image and color histogram of the retrieval object image is smaller than a certain 
threshold value, the outputting process S1106 for outputting the retrieval object 
image (multimedia data to be retrieved) as the similar image is abridged. 

After that, in the similarity comparing process, it is judged whether the 
comparing process is performed up to the last retrieval object image (Image N) 
10 S1107. 

When the comparing process for comparing the similarity to the last 
retrieval object image (Image N) is not finished, it is judged whether the histogram 
construction about the next retrieval object image inside of the retrieval object 
system 1003 is differentiated from former S1109. 

15 When the histogram construction about the next retrieval object image is 

differentiated from the former, and the processes from the acquiring process 
S1101 for acquiring the color space description information and quantization 
description information of the retrieval object image are performed repeatedly. 

On the contrary, when the histogram construction about the next retrieval 

20 object image is same with the former, the processes from the reading process 
S1103 for reading the color histogram values 1008-1 ~ 1008-N about the retrieval 
object image are performed repeatedly. 

Meanwhile, when the process for comparing the similarity is performed up 
to the last retrieval object Image (Image N), it is judged whether the other query 

25 image is inputted S1 108. 
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When the other query image is Inputted, the processes from the inputting 
process S1100 for inputting the query image are performed repeatedly, when the 
other query image is not inputted, the multimedia data retrieval is finished. 

Hereinafter, a third embodiment of the content-based multimedia data 

5 retrieval method performed by the system of FIG.10 will now be described in detail 
with reference to accompanying FIG. 12. 

FIG. 12 is a flow chart illustrating the third embodiment of the content- 
based multimedia retrieval method In aecordance with the present invention. The 
processs for storing the color histogram of the query image and the process for 

10 using the stored color histogram to the multimedia retrieval are same with the 
second embodiment of FIG. 11. Accordingly, hereinafter, the processes S1100, 
S1101, S1103 ~ S1109 same with FIG. 11 are described with same reference 
numerals, and overlapped descriptions are abridged. 

First, when the query image is Inputted S1100, by referencing the color 

15 space and color quantization description information of the retrieval object Image 
S1101, It is judged whether the color histogram same with the inputted query 
image is quantized so as to correspond to the color histogram construction of the 
retrieval object image, is extracted, and is stored In the color histogram database 
1001 of the query Image SI 200. 

20 When the color histogram of the query image is extracted before and is 

stored In the color histogram database 1001 of the query image, the color 
histogram value of the stored query Image is read SI 202, and the processes from 
the reading process S1 103 for reading the color histograms values 1008-1 ~ 1008- 
N about the retrieval object image are performed. 

25 On the contrary, when the color histogram of the query Image is not stored 
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in the color histogram database 1001 of the query image after extracting before, 
the color histogram of the inputted query image is quantized with the color space 
and quantization method corresponding to the color histogram construction of the 
retrieval object image, and is extracted S1201. 
5 After that, the color histogram of the extracted query image is stored in the 

color histogram database 1001 of the query image S1203, the processes from the 
reading process S1 103 for reading the color histograms values 1008-1 ~ 1008-N 
about the retrieval object image are performed. 

As described above, since the color histogram of the query image 

10 corresponding to the color histogram construction of the image to be retrieved is 
stored in advance, the present invention can skip a new process for extracting the 
color histogram of input query image when a query image same as the stored 
color histogram of the query image is inputted next time. Accordingly, the present 
invention can improve the multimedia retrieval speed. 

15 As described above, the present invention is capable of performing an 

interoperable multimedia retrieval between different systems using the color 
histogram constructed with different color spaces and color quantization methods. 

In addition, the present invention is capable of retrieving an image to be 
retrieved corresponding to the query image of itself in a database of the other party 

20 when retrieving multimedia data in the systems using different color histogram 
constructed with different color spaces and color quantization methods. 

In addition, the present invention is capable of retrieving images by 
connecting databases storing various and different and wide range of images on a 
network environment such as Internet. 

25 In addition, the present invention is capable of improving the multimedia 
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retrieval speed by extracting and storing the color histogram of the query image in 
advance. 



